Exploring Distributed Resource Allocation Techniques in the SLURM Job Management System

With the exponentially growth of distributed computing systems in both flops and cores, scientific applications are growing more diverse with a variety of workloads. These workloads include traditional large-scale High Performance Computing MPI jobs, and ensemble workloads, such as Many-Task Computing workloads comprised of extremely large number of tasks of finer granularity, where tasks are defined on a per-core or per-node level, and often execute in milliseconds to seconds. Delivering high throughput and low latency for these heterogeneous workloads requires developing distributed job management system that is magnitudes more scalable and available than today’s centralized batch-scheduled job management systems. In this paper, we present a distributed job launch prototype SLURM++, which extends the SLURM resource manager by integrating the ZHT zero-hop distributed key-value store for distributed state management. SLURM++ is comprised of multiple controllers with each one managing several SLURM daemons, while ZHT is used to store all the job metadata and the SLURM daemons’ state. We compared SLURM with our SLURM++ prototype with a variety of microbenchmarks of different job sizes (small, medium, and large) at modest scales (500-nodes) with excellent results (10X higher job throughput). Scalability trends shows expected performance to be many orders of magnitude higher at tomorrow’s extreme scale systems. Keywords-job management systems; job launch; distributed scheduling; key-value stores